WebSocket mode

WebSocket mode is the bi-directional form of the Bot API. Live Hub opens one WebSocket per call, and both sides send messages over it whenever they have something to say. A voice bot depends on that freedom, which is why LLM agents generally use this mode. For the request-response form, see HTTP mode.

Unlike HTTP mode, WebSocket mode can carry audio in both directions, so a bot that performs its own speech recognition or its own synthesis can exchange the audio directly.

The WebSocket connection

Live Hub opens the connection at the start of the call, to the URL configured on the bot connection, using an HTTP Upgrade request. It stays open for the whole conversation and serves exactly one conversational session.

Two proprietary headers on the handshake give you the call's context before any message arrives:

Header	What it carries
`AC-Caller-Number`	The caller's phone number, the same value as `caller` in session.initiate.
`AC-Conversation-Id`	The conversation's unique identifier, the same value as `conversationId`.

Authentication

Live Hub authenticates on the handshake, in the Authorization header of the Upgrade request, with either a permanent bearer token or an OAuth 2.0 token. Your server checks it there and accepts or rejects the connection. See Security and authentication.

GET /bot/ws HTTP/1.1
Host: bot.example.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Version: 13
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Authorization: Bearer {token}
AC-Caller-Number: +1234567890
AC-Conversation-Id: 4a5b4b9d-dab7-42d0-a977-6740c9349588

Message format

Every message is JSON, with audio base64-encoded inside it rather than sent as binary frames. Each one carries a type naming the message. Messages traveling from Live Hub to the bot also carry conversationId.

Configuration

On the bot connection, set 'Bot connection API type' to WebSocket mode and enter your service's WebSocket address in the 'Bot URL' field, the botURL parameter. Select an 'Authentication method' on the same screen. See AudioCodes Bot API.

To have audio pass directly between the caller and the bot rather than through Live Hub's speech engines, select the 'Enable voice streaming' check box on the bot connection. The underlying parameters are directSTT, which streams the caller's voice to the bot with no speech-to-text in between, and directTTS, which plays the bot's audio to the caller with no text-to-speech.

Messages from Live Hub

Live Hub sends the following messages to the bot.

session.initiate

Live Hub sends session.initiate once the session is established. Answer with session.accepted, or with session.error to decline the conversation.

Parameter	Type	Description
`conversationId`	string	Unique identifier for the conversation.
`type`	string	`session.initiate`.
`botName`	string	The configured name of the bot.
`caller`	string	The caller's phone number.
`expectAudioMessages`	boolean	`true` when the bot is expected to play audio itself, in which case it must not send `message` activities carrying text prompts. It follows the `directTTS` parameter and nothing else.
`supportedMediaFormats`	array	The audio coders Live Hub accepts, most preferred first.

The media formats are:

Format	Encoding
`raw/mulaw`	Mu-Law, 8-bit 8 kHz, no header
`wav/mulaw`	Mu-Law, 8-bit 8 kHz, WAV header
`raw/lpcm16`	Linear PCM, 16-bit 16 kHz, no header
`wav/lpcm16`	Linear PCM, 16-bit 16 kHz, WAV header
`raw/lpcm16_8`	Linear PCM, 16-bit 8 kHz, no header
`wav/lpcm16_8`	Linear PCM, 16-bit 8 kHz, WAV header
`raw/lpcm16_24`	Linear PCM, 16-bit 24 kHz, no header
`wav/lpcm16_24`	Linear PCM, 16-bit 24 kHz, WAV header

{
  "conversationId": "4a5b4b9d-dab7-42d0-a977-6740c9349588",
  "type": "session.initiate",
  "botName": "my_bot_name",
  "caller": "+1234567890",
  "expectAudioMessages": true,
  "supportedMediaFormats": [
    "raw/lpcm16"
  ]
}

activities

The activities message carries a list of activities. Two of them are shown here.

The start event arrives when the call begins:

{
  "conversationId": "4a5b4b9d-dab7-42d0-a977-6740c9349588",
  "type": "activities",
  "activities": [
    {
      "type": "event",
      "name": "start",
      "id": "582bbc43-0ef7-47e9-97b4-1e6141625b01",
      "timestamp": "2022-07-20T07:15:48.239Z",
      "language": "en-US",
      "parameters": {
        "locale": "en-US",
        "caller": "caller-id",
        "callee": "my_bot_name"
      }
    }
  ]
}

The dtmf event arrives when the caller presses a key:

{
  "conversationId": "4a5b4b9d-dab7-42d0-a977-6740c9349588",
  "type": "activities",
  "activities": [
    {
      "type": "event",
      "name": "dtmf",
      "id": "582bbc43-0ef7-47e9-97b4-1e6141625b01",
      "timestamp": "2022-07-20T07:15:48.239Z",
      "language": "en-US",
      "value": "123"
    }
  ]
}

userStream.start

userStream.start asks to start streaming the caller's audio to the bot. Answer with userStream.started; Live Hub begins sending userStream.chunk messages once it arrives.

{
  "conversationId": "4a5b4b9d-dab7-42d0-a977-6740c9349588",
  "type": "userStream.start"
}

userStream.chunk

userStream.chunk carries one chunk of the caller's audio.

{
  "conversationId": "4a5b4b9d-dab7-42d0-a977-6740c9349588",
  "type": "userStream.chunk",
  "audioChunk": "Base64EncodedAudioData"
}

userStream.stop

userStream.stop ends the audio stream. Answer with userStream.stopped.

{
  "conversationId": "4a5b4b9d-dab7-42d0-a977-6740c9349588",
  "type": "userStream.stop"
}

session.resume

Live Hub sends session.resume when it reconnects after losing the WebSocket. Answer with session.accepted, or with session.error to decline the reconnection.

{
  "conversationId": "4a5b4b9d-dab7-42d0-a977-6740c9349588",
  "type": "session.resume"
}

session.end

session.end reports that the conversation is over.

{
  "conversationId": "4a5b4b9d-dab7-42d0-a977-6740c9349588",
  "type": "session.end",
  "reasonCode": "client-disconnected",
  "reason": "Client Side"
}

Messages from the bot

The bot sends the following messages to Live Hub.

session.accepted

session.accepted accepts a session.initiate or a session.resume.

Parameter	Type	Description
`mediaFormat`	string	The format you have chosen. It must be one of the `supportedMediaFormats` from `session.initiate`.

{
  "type": "session.accepted",
  "mediaFormat": "raw/lpcm16"
}

userStream.started

userStream.started confirms that you are ready for audio chunks.

{
  "type": "userStream.started"
}

userStream.stopped

userStream.stopped confirms that you will accept no more audio chunks.

{
  "type": "userStream.stopped"
}

userStream.speech.hypothesis

userStream.speech.hypothesis carries a partial recognition result. Send these messages: Live Hub uses them to drive barge-in.

Parameter	Type	Description
`alternatives`	array	The recognition alternatives, each with a `text`.

{
  "type": "userStream.speech.hypothesis",
  "alternatives": [
    {
      "text": "How are"
    }
  ]
}

userStream.speech.recognition

userStream.speech.recognition carries the final recognition result. It is mainly useful for logging.

Parameter	Type	Description
`alternatives`	array	The recognition alternatives, each with a `text` and a `confidence` between 0 and 1.

{
  "type": "userStream.speech.recognition",
  "alternatives": [
    {
      "text": "How are you.",
      "confidence": 0.83
    }
  ]
}

userStream.speech.started

userStream.speech.started reports that the bot detected the start of speech. It applies when the bot does its own voice activity detection. With the OpenAI Realtime API, for example, it maps to input_audio_buffer.speech_started. Send it: Live Hub uses it for barge-in and timeouts.

{
  "type": "userStream.speech.started"
}

userStream.speech.stopped

userStream.speech.stopped reports that the bot detected the end of speech; it maps to input_audio_buffer.speech_stopped on the OpenAI Realtime API. Send it: Live Hub uses it for barge-in and timeouts.

{
  "type": "userStream.speech.stopped"
}

userStream.speech.committed

userStream.speech.committed reports that the bot has committed the caller's speech for processing, normally at the end of an utterance; it maps to input_audio_buffer.committed on the OpenAI Realtime API. Send it: Live Hub uses it for barge-in and timeouts.

It usually follows a single userStream.speech.stopped, but not always: the bot may wait a moment to be sure the utterance is finished, so there can be a delay, or several userStream.speech.stopped messages before one userStream.speech.committed.

{
  "type": "userStream.speech.committed"
}

playStream.start

playStream.start opens a play stream, which is how the bot streams audio to the caller. Follow it with playStream.chunk messages, and close it with playStream.stop.

Parameter	Type	Description
`streamId`	string	An identifier for the stream, unique within the conversation.
`mediaFormat`	string	The stream's media format. It must be one of the values from `session.initiate`.
`altText`	string	Optional. Text to log in place of the audio.
`activityParams`	object	Optional. Further activity parameters, such as `expectAnotherBotMessage`.

Only one play stream can be open at a time. Close the current one with playStream.stop before starting another.

A play stream is for audio the bot produces itself. To have Live Hub speak text instead, send a message activity, which is available only when text-to-speech is configured. To play a pre-recorded file or buffer, use the playUrl activity.

{
  "type": "playStream.start",
  "streamId": "1",
  "mediaFormat": "raw/lpcm16"
}

playStream.chunk

playStream.chunk carries one chunk of audio for the open play stream. Chunks are accepted only while the stream is open.

Parameter	Type	Description
`streamId`	string	The stream's identifier.
`audioChunk`	string	The audio, base64-encoded.

Send audio at the rate it plays back. If you send it faster or slower, the caller hears the difference.

{
  "type": "playStream.chunk",
  "streamId": "1",
  "audioChunk": "Base64EncodedAudioData"
}

playStream.stop

playStream.stop closes the play stream.

Parameter	Type	Description
`streamId`	string	The stream's identifier.

{
  "type": "playStream.stop",
  "streamId": "1"
}

activities

The activities message carries a list of activities from the bot. You use it for two things in particular.

Play an audio buffer to the caller with a playUrl activity. Put the audio in playUrlUrl as a data URI: base64, prefixed data:audio/wav;base64 or data:application/octet-stream;base64 depending on whether it has a WAV header. playUrlMediaFormat gives the format, and the optional playUrlAltText supplies the corresponding text.

{
  "type": "activities",
  "activities": [
    {
      "type": "event",
      "name": "playUrl",
      "activityParams": {
        "playUrlAltText": "Welcome to our example",
        "playUrlUrl": "data:audio/wav;base64,UklGRmK4AABXQVZFZm10IBIAAAAGAA...",
        "playUrlMediaFormat": "wav/lpcm16"
      }
    }
  ]
}

Disconnect the call with a hangup activity.

Closing the WebSocket does not end the call: Live Hub treats that as a dropped connection and tries to reconnect. Only a hangup activity ends it: Live Hub responds with session.end, disconnects the call, and then closes the WebSocket.

{
  "type": "activities",
  "activities": [
    {
      "type": "event",
      "name": "hangup"
    }
  ]
}

session.error

session.error reports a fatal error. Live Hub disconnects the call and closes the WebSocket.

Parameter	Type	Description
`reason`	string	The error message.

{
  "type": "session.error",
  "reason": "Internal Server Error"
}

Agent assist

WebSocket mode supports assist bots natively, by marking which participant a message concerns.

After the call starts, send a startRecognition activity naming the participant whose audio you want to receive:

{
  "type": "activities",
  "activities": [
    {
      "type": "event",
      "name": "startRecognition",
      "activityParams": {
        "targetParticipant": "customer"
      }
    }
  ]
}

On an agent-assist call, Live Hub adds a participant parameter to userStream.start, userStream.chunk, and userStream.stop. Your bot must include the same parameter on userStream.started, userStream.stopped, userStream.speech.hypothesis, and userStream.speech.recognition.

Check connectivity

Two messages verify the connection during integration, and they are never part of a real call. The Validate bot connection configuration button on the bot connection uses them.

Live Hub sends:

{
  "type": "connection.validate"
}

Reply with:

{
  "type": "connection.validated",
  "success": true
}

Example call flow

The following diagram traces a complete call, from session.initiate through audio streaming to session.end.

A complete call, from session.initiate through audio streaming to session.end